Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.

Temporal knowledge graph question answering (TKGQA) poses a significant challenge task, due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. Although large language models (LLMs) have made considerable progress in their reasoning ability over structured data, their application to the TKGQA task is a relatively unexplored area. This paper first proposes a novel generative temporal knowledge graph question answering framework, GenTKGQA, which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. First, we exploit LLM's intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. Next, we design virtual knowledge indicators to fuse the graph neural network signals of the subgraph and the text representations of the LLM in a non-shallow way, which helps the open-source LLM deeply understand the temporal order and structural dependencies among the retrieved facts through instruction tuning. Experimental results demonstrate that our model outperforms state-of-the-art baselines, even achieving 100\% on the metrics for the simple question type.

Large language models (LLMs) have demonstrated impressive reasoning abilities in complex tasks. However, they lack up-to-date knowledge and experience hallucinations during reasoning, which can lead to incorrect reasoning processes and diminish their performance and trustworthiness. Knowledge graphs (KGs), which capture vast amounts of facts in a structured format, offer a reliable source of knowledge for reasoning. Nevertheless, existing KG-based LLM reasoning methods only treat KGs as factual knowledge bases and overlook the importance of their structural information for reasoning. In this paper, we propose a novel method called reasoning on graphs (RoG) that synergizes LLMs with KGs to enable faithful and interpretable reasoning. Specifically, we present a planning-retrieval-reasoning framework, where RoG first generates relation paths grounded by KGs as faithful plans. These plans are then used to retrieve valid reasoning paths from the KGs for LLMs to conduct faithful reasoning. Furthermore, RoG not only distills knowledge from KGs to improve the reasoning ability of LLMs through training but also allows seamless integration with any arbitrary LLMs during inference. Extensive experiments on two benchmark KGQA datasets demonstrate that RoG achieves state-of-the-art performance on KG reasoning tasks and generates faithful and interpretable reasoning results.

Ontological knowledge, which comprises classes and properties and their relationships, is integral to world knowledge. It is significant to explore whether Pretrained Language Models (PLMs) know and understand such knowledge. However, existing PLM-probing studies focus mainly on factual knowledge, lacking a systematic probing of ontological knowledge. In this paper, we focus on probing whether PLMs store ontological knowledge and have a semantic understanding of the knowledge rather than rote memorization of the surface form. To probe whether PLMs know ontological knowledge, we investigate how well PLMs memorize: (1) types of entities; (2) hierarchical relationships among classes and properties, e.g., Person is a subclass of Animal and Member of Sports Team is a subproperty of Member of ; (3) domain and range constraints of properties, e.g., the subject of Member of Sports Team should be a Person and the object should be a Sports Team. To further probe whether PLMs truly understand ontological knowledge beyond memorization, we comprehensively study whether they can reliably perform logical reasoning with given knowledge according to ontological entailment rules. Our probing results show that PLMs can memorize certain ontological knowledge and utilize implicit knowledge in reasoning. However, both the memorizing and reasoning performances are less than perfect, indicating incomplete knowledge and understanding.

Americans have plenty of negative opinions about artificial intelligence - but has anyone ever stopped to think: 'What do the machines think about Americans?' Polls show that a majority worry advanced AI will become a'threat to the human race' (57 percent), half consider self-driving cars dangerous, and more than half (54 per cent) believe AI will play a role in America's decline over the coming decades. The feeling might be mutual -- judging by the responses that ChatGPT and the image-generator Midjourney gave DailyMail.com ChatGPT stated that people in Alabama are'hillbillies', Idahoans are'gun-toting survivalists', Wisconsinites are'heavy drinkers' and the citizens of Iowa are just plain'boring'. The AI could not think of anything bad to say about'friendly' Nebraskans, however. While not all 50 US states were as easy for Midjourney to caricature as they were for ChatGPT, the image-maker did manage to roast the citizens of states with notorious or outsized reputations, like California and New Jersey.

Artificial intelligence may not be sentient yet, but complex machine-learning algorithms like ChatGPT are slowly but surely developing their own opinions and prejudices about us. Midjourney, an AI program that creates images from textual descriptions, has some unusual and sometimes comically simplistic ideas on what makes the quintessential American resident of each US state. Now in its fifth version, Midjourney's inner workings are a closely held trade secret of the non-profit research lab that created the AI, unlike its text-to-image competitors DALL-E and Stable Diffusion, which use an open-source model. Each text-to-image request yielded four images from which we selected the most accurate or - failing that - the most subjectively interesting. The AI-generated images of men from D.C. were also ostentatiously dressed for politics and punditry Midjourney appears to intuitively believe that the vast majority of women in our nation's capitol skew older.

Most weakly supervised named entity recognition (NER) models rely on domain-specific dictionaries provided by experts. This approach is infeasible in many domains where dictionaries do not exist. While a phrase retrieval model was used to construct pseudo-dictionaries with entities retrieved from Wikipedia automatically in a recent study, these dictionaries often have limited coverage because the retriever is likely to retrieve popular entities rather than rare ones. In this study, we present a novel framework, HighGEN, that generates NER datasets with high-coverage pseudo-dictionaries. Specifically, we create entity-rich dictionaries with a novel search method, called phrase embedding search, which encourages the retriever to search a space densely populated with various entities. In addition, we use a new verification process based on the embedding distance between candidate entity mentions and entity types to reduce the false-positive noise in weak labels generated by high-coverage dictionaries. We demonstrate that HighGEN outperforms the previous best model by an average F1 score of 4.7 across five NER benchmark datasets.

As stated correctly by Dr. Lynn Lashbrook of Sports Management Worldwide, "the frontier of analytics is just beginning, and there is no end in sight to the potential." As data continues to play a crucial role in practically every industry, it is no surprise that we are seeing the emergence of sports analytics. With popular sports like soccer, athletics, and tennis being watched by millions across the globe, the worldwide market size for sports analytics is projected to reach $3.44 billion by 2028. Looking at the latest players in this market, the Real Madrid football club is using analytics tools to manage and improve relationships with over 450 million fans. Effectively, sports analytics is being leveraged for both on-field analytics (for improving player performance & team strategies) and for off-field analytics (fan engagement, merchandise sales).

Knowledge Bases (KBs) provide structured representation of the real-world in the form of extensive collections of facts about real-world entities, their properties and relationships. They are ubiquitous in large-scale intelligent systems that exploit structured information such as in tasks like structured search, question answering and reasoning, and hence their data quality becomes paramount. The inevitability of change in the real-world, brings us to a central property of KBs -- they are highly dynamic in that the information they contain are constantly subject to change. In other words, KBs are unstable. In this paper, we investigate the notion of KB stability, specifically, the problem of KBs changing due to real-world change. Some entity-property-pairs do not undergo change in reality anymore (e.g., Einstein-children or Tesla-founders), while others might well change in the future (e.g., Tesla-board member or Ronaldo-occupation as of 2022). This notion of real-world grounded change is different from other changes that affect the data only, notably correction and delayed insertion, which have received attention in data cleaning, vandalism detection, and completeness estimation already. To analyze KB stability, we proceed in three steps. (1) We present heuristics to delineate changes due to world evolution from delayed completions and corrections, and use these to study the real-world evolution behaviour of diverse Wikidata domains, finding a high skew in terms of properties. (2) We evaluate heuristics to identify entities and properties likely to not change due to real-world change, and filter inherently stable entities and properties. (3) We evaluate the possibility of predicting stability post-hoc, specifically predicting change in a property of an entity, finding that this is possible with up to 83% F1 score, on a balanced binary stability prediction task.

We are living in a world that is constantly evolving. In general, every industry is facing new innovative changes thanks to Artificial Intelligence (AI). People are taking advantage of AI in order to find new ways that will simplify their daily life. We can notice that there have been advancements in almost every industry, however, the industry that made the biggest step forward and improved itself thanks to AI is definitely the sports betting one. From a growth perspective, we can freely say that AI already plays a significant role in sports, and this is a clear sign that it will change it even more in the future.